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Using a Markov chain model for the motion of a particle through 
a V-node network, we consider the quantities ny, which are the 
average number of steps taken by the particle in traveling from an 
originating node, i, to a destination node, j. A figure-of-merit, N, for 
the entire network is introduced by averaging ny over i and j. We 
investigate which networks minimize or maximize N, either when no 
restriction is placed on the Markov chain, or when we restrict it so 
that it corresponds to random routing. By the latter we mean that at 
each node the particle "selects at random" lines from an undirected 
network graph. We show that for random routing, the complete graph 
has N= (V- 1) and is the minimizing graph. The maximizing graph 
is unknown, but we establish that the worst behavior of N increases 
at least with the cube of the number of vertices, but no worse than the 
3.5 power. Properties of the class of graphs known as barbells are 
useful here. The minimizing unrestricted chain corresponds to plac- 
ing the nodes on a circle and proceeding unidirectionally from one 
node to the next. Here, N = V/2. 

I. INTRODUCTION AND RESULTS 

Our work is set against the background of the Markov chain model 
for the movement of a "particle" or "message" through a network of 
V nodes. Thus, suppose a particle originates at node i and is destined 
for node j ^ i. The particle wanders through the network toward its 
destination via a Markov chain, going to node n from node m with 
probability pmn. The quantity p mn is the (m, n) element of the transition 
matrix P of the Markov chain whose states correspond to the nodes of 
the network. Denote the average number of steps required by the 
particle to reach its destination by n tj , and assume that any node is 
accessible from any other (the Markov chain is irreducible). Then we 
introduce the figure-of-merit, N, for any such chain by averaging n i} 
over the (V - 1) possible destinations and V origins of particles: 
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Our problem is: Which chains minimize or maximize N, either for a 
random-walk Markov chain or for an unrestricted chain? 

An unrestricted chain means that no restrictions are placed on the 
transition matrix P other than irreducibility and the requirement that 
pa ■ for all i. The random-walk chain is a special case of interest and 
is defined as follows. Draw any connected undirected graph on the V 
nodes. If m and n are not joined by an edge of this graph set pmn = 0, 
while if they are so joined set pmn = l/l m , where l m is the number of 
edges of the graph leaving node m. Thus, at any node, the particle 
chooses from the available lines "at random." This random walk on 
the graph has previously been used by Kleinrock in Ref. 1, where it is 
referred to as random routing. 

Our results are: 
(i) For random routing, the average number of steps N is mini- 
mized when the graph of the network is the complete graph. For this 
caseN=(V-l). 

(ii) For the unrestricted chain, N is minimized when the nodes are 
placed consecutively on a circle and we proceed deterministically from 
1 to 2 to 3, etc. Here N = V/2. 

(Hi) By choosing pn = P21 = 1 — e, e — » 0, N can be arbitrarily large 
for the unrestricted chain (when V> 2). 

(iv) We have not been able to determine the "worst" graph for 
random routing, but we can show for large V that 0(V 3 ) < N woret < 
OiV 35 ). The barbell graphs of Mitra- Weiss 2 and Landau-Odlyzko 3 are 
good candidates for bad graphs. 

II. THE MINIMAL WALK 

In this section, we demonstrate that the complete graph is the only 
graph which minimizes N for the random-walk problem. 

The fact that the symmetry of the complete graph requires ny = N 
for all i 7* j allows a simple demonstration of the fact N = ( V — 1) for 
this case. If the particle originates at node i, we go directly to our 
destination,/ with probability l/( V — 1), requiring only one step, or we 
go to another node with probability (V — 2)/( V — 1) and then require 
an average of (1 + TV) steps to reach j from i. Thus, 

1 V-2 

N= v=T + v=T [1 + Nl (2) 

Solving (2) yields N = (V - 1). 
To show N > (V — 1) for other graphs is more involved. We first 
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give properties of those transition matrices that correspond to random 
routing, ending with (12) which gives the stationary probabilities for 
those chains. We then derive (24), which is an expression for the 
average of the first passage times with which we are concerned. Finally, 
we obtain our result by giving a lower bound to (24). 

Standard results on Markov chains or positive matrices may be used 
without reference when needed. For the former, the reader may consult 
Ref. 4, while Ref. 6 is a useful source for matrices. 

In this paper, we denote transposition, complex conjugation, and 
hermitian conjugate by the symbols T, *, and + , respectively. 

For random routing, the transition matrix P is given by 

P = DA, (3) 

where A is the symmetric adjacency matrix of the graph and is defined 
by 

1 if i ¥* j and i and /' are joined 
a.ij = by an edge 

otherwise , 

and D is a diagonal matrix with diagonal elements 

da = 1/k, (4) 

/, being the ith row sum of A and also, therefore, the number of edges 
of the graph incident on node i. The matrix P is stochastic, that is, it 
has nonnegative elements and the rows sum to one. Further, the 
assumed connectivity of the network graph implies that P is irreduci- 
ble. The facts that P is stochastic and irreducible imply that the largest 
positive eigenvalue of P is unity and has multiplicity one. All other 
eigenvalues of P have modulus less than, or equal to, one. Set Ai = 1 
and let A 2 , • • ■ , A v be the remaining eigenvalues of P. 

We now investigate the eigenvalue and eigenvector structure of P. 
The matrices P = DA and Q ' = D 1/2 AD 1/2 differ by a similarity 
transformation (P = D y2 QD~ 1/2 ) and so have the same eigenvalues. 
Since Q is real symmetric, the A, are real. Further Q has a complete set 
of orthonormal eigenvectors </> 0) 

Q* li) = A^" 

*»V#J = S*. (5) 

If we denote the eigenvectors of P and P T , which correspond to A, by 
U'" and W ( '\ respectively, then we clearly have 

u (,) = «z> v V° 
w (I '> = ar 1 zr 1/ V ,r 

W li)T V U) = Sij. (6) 

MARKOV CHAINS 2067 



The a, in (6) are any convenient constants. In addition, we have the 
spectral representation 

v 
P = £ AiU w W W1 '. (7) 

i-l 

Since the rows of P sum to unity, the eigenvector U (1) may be chosen 
to be 





(8) 

v/ 

The eigenvector W (1) , 

P^W (1) = W (1) , (9) 

is then the stationary probability vector for the Markov chain 

(10) 



obeying the normalization of (6). 

Equation (9) is easy to solve when P corresponds to random routing. 
Define a vector Y by (Y), = U, so DY = U (1) . Then, using (3), 

P T Y = ADY = AXJ a) = Y. (11) 

Thus, the stationary probability vector for the chain has components 

A = 2l>0, (12) 

where L equals the number of edges in the graph. Using (12) in (6), we 
find 

(♦«), = y/pi. (13) 

We next derive the expression (24) for N that will be our point of 
departure. Let fa(n), n = 1, 2, • • • be the first passage probability from 
node i to nodey at time n. Then, if P" is the nth power of the transition 
matrix P, we have (Feller, in Ref. 4, p. 352) 

ft(D = Pij (14) 

fij(n) = (P n )ij - J (P k )jjfij(n -k), n>2. 
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In terms of the generating functions, defined for | s | < 1 by 

Fij(s) = l fij(n)s n 



Pij(s) = S (P n )ijs n , 



(14) is equivalent to 



Fij(s) = 



Pij(s) 



\s\<l. 



(15) 
(16) 

(17) 



1 + Pjj(s)' 

Since all eigenvalues A, of P satisfy | A, | < 1, (16) shows that (17) may 
be rewritten for i ¥* j as 

FtfW -[J-«P]^ |s|<l- (18) 

Denoting the ith component of U ( ' i) by t7|' ,) J and similarly for W*' 1 *, 
set 

t/^»Wy = bf (19) 

and note from (8) and (10) that 

bf= Pj . (20) 

Then, using the spectral representation (7), (18) may be written 

pj+a-s) z-J—btf 

FtAs) = "7 21 A • (2D 



Pj + a - s) 2 



=2 1 - sA, 



bW 



M=2 



In this form, F,y(s) may be analytically continued to a neighborhood of 
s = 1. Then, from (15) and a standard Abelian theorem we have 



»«= I nfij(n) = 



gV(g = 1) 
ds 



Using (21) to (22), we obtain 

1 ^ 



» 



Pj u-2 1 - A„ 



ibf-bf). 



(22) 



(23) 



Finally, using the definition (1) and relations (6) and (19), we see that 
the average number of steps, N, required to reach a destination with 
random routing for V nodes may be written 



1 v 1 



' v i i 



W 



7 Vpy 



(24) 
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Again,/), are the stationary probabilities of the Markov chain governed 
by transition matrix P — DA. The A M , fi > 2, are the nonunity 
eigenvalues of P and Q = D 1/2 AD 1/2 , the latter matrix having ortho- 
normal eigenvectors 4> (/l) . Also 4> (1) , the eigenvector of Q associated with 
the eigenvalue Ai = 1, is given by (13). 

We now lower bound the right member of (24). A few known facts 
about the quantities A M , $ {fl) , and pj will be exploited, but their great 
interdependence (they are all determined by P) will be ignored. 

Set 



v 1 1 

y=i Pj V 



w>> 2 
j ylpj 



and rewrite (24) as 



ir-JUz « 



V-l^al-V 



We assert a M > 0, since, by Cauchy's inequahty, 






s lH? 



1*7 



(M)|2 



SI 
7 



j 



i*y 



0O|2 



A 



(25) 



(26) 



(27) 



However, equality can hold only if 



(M) 



AM 



<*i, 



that is, if 



*r <*&-*?. 



Since </> (m) is orthogonal to <f> (1) for /x > 2, this cannot happen.* 

With the positivity of the a^ established, niinimize (26) over the A^ 
holding the a ll > Q fixed. During this minimization, we only impose 
two constraints on the A <M) . First, A M < 1, and second £jf_2 A M = — 1. The 
first has been amply discussed already while the second follows from 
the fact that the diagonal elements of the transition matrix P are zero 
and so 



= tr P - £ \ 



(28) 



M-i 



Since Ai = 1, the second constraint follows. Introducing the second 
constraint via a Lagrange multiplier (call it — /?), we find that the 
unique stationary point of 



incidentally, if we write n,> = £,, t{JV(1 — A^) it is not necessary that -r-f > 0. 
However, r\f + rff > 0. 
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£-\--/?2A M , (29) 

satisfying the two constraints, occurs when 

(30) 




Therefore, applying (30) to (26) gives 



N *voh»Hrf- (31) 

Next, use the fact that the arithmetic mean exceeds the geometric 
mean: 

2 



jv^- 1 






V -\ <v •■ x n 



^r 1 



^(M''' <32) 

Motivated by the definition of the <v [see (25)], introduce the real 
symmetric matrix M having elements 

Mf-ia*-! (33) 

Pi V Jptfj 

Then, 

a^-^W - (34) 

Note that 

M *<» = i (- 8, - t,-F=) ^ = °' (35) 

/-i \Pf v y/piPj/ 

while we have already shown 

4>*M<t>>0 if (4>, <fr (1) ) = 0. (36) 

Thus, Mis positive semi-definite, having precisely one zero eigenvalue. 
Fix p, > 0, i = 1, • • •, V, and proceed to lower bound (32) by writing 

v v 

[I a M > min n ^W, ( 37 ) 

M -2 »-2 

the minimum in (37) being taken over all orthonormal sets of vectors 
ip lv) which satisfy 
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(i|/"\ <f> m ) - 0, fm>-S,...,V. (38) 

An inequality of Ky Fan 6 implies that under the conditions of (38) we 
have 

v 
min n ^'W = M1M2, • • •, jUv-i, (39) 

where ju„ i = 1, • • • , V — 1 are the nonzero eigenvalues of the matrix M, 
which was defined in (33). Thus, 

V-l — 

N>—y-[ [ i lf i 2 ... [ xv-i] v - 1 . (40) 

If gi are the components of an eigenvector of M associated with 
eigenvalue fi, the equation 

£jJto-/vi (4i) 

yields, when (33) is substituted for My, 

gi = k -i -, (42) 



<a..y 



A being a normalization constant. Since for /t ¥= we must have 

O-Xrf-Z^, (43) 

we obtain, using (42), 

1 1 

= 0. (44) 



w/1 

M 

Equation (44) determines the (V— 1) positive eigenvalues /*,-. Clearing 
fractions in (44) yields 

^n^-^O (45) 

with 

B -a(H' M6) 

The denominator D in (45) may be discarded and then the product of 
the nonzero eigenvalues of M is simply read off the remaining poly- 
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nomial. Since the coefficient of /i V ! in the numerator polynomial in 
(45) is V(-l) v -\ 



1 v v 1 
Mi • • • pv-i =yl n -• 



(47) 



/Vi 



The product of positive numbers having a fixed sum is maximized 
when each number is the same, so we have 



v 

/Vi 



' a- p*) 
v-i 



V-l 



or 



» 1 y( v ~ 1 



V-l 



(48) 



Finally, the minimum of (48) subject to J]*-i Pi = 1 occurs when p, = 

1 
y' or ' 




v-i 



(49) 



Combining (49) with (40) produces our desired result: 

(V-l). 



V-l -L 



(50) 



It is easy to work backwards through the argument to see that the 
complete graph is the only one that can achieve N= (V-l). Clearly, 
equality in the last step of the argument can only be attained if p, = 

- . Substituting this into (24) and using the orthonormality of the 
^ yields 



V-l v 1 

n = - — - y — - — , 
v ,%1-x; 

which only equals the minimum when X M = — 
Using this, and the fact that Ai = 1 in (7) yields 



V-l 



(51) 



, M = 2, •••, V 



P = U (1) W (1)r - 

1 



1 y v U) w li)T 

v - 1 ti 



V-lit 



X u" , w ,,,T + -J— u a, w (ur 



(52) 



V-l 
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The first dyad sum in the right member of (52) is the identity matrix, 
while the last dyad in (52) is the matrix that has all elements equal to 
1/V. Thus, P is precisely the transition matrix associated with the 
complete graph. 

III. THE MINIMAL CHAIN 

Now we turn to rinding the minimum value of N for the unrestricted 
chain. In the introduction we stated that for this we should place the 
nodes on a circle and go unidirectionally from one node to the next. 
Clearly, in this case, Yj n u does not depend on i, and, in fact, 

N = -^—l m,=-i-[l + 2 + 3+... +<V-D]--£ (53) 

V — 1 y _2 V 1 A 

We must now show that no other setup can do as well. For this end, 
expressions such as (23) are not useful since the \,, U (m) and W <M) may 
be complex and it would be difficult to pick out even real combinations, 
let alone positive ones that might be lower bounded. Rather, we retreat 
to an obvious generalization of the argument we used to derive (2). 
This generalization reads 

nij =Pu + £ Pad + n kj ), ){~. *' ' "' V (54) 

k J V l 

or, since P is stochastic, 

n*- Elton* -1, l ; J J = - l, '" ,V - (55) 

Equation (55) is our new point of departure. 

Let N ( -' ) , j = 1, • • ., V, be the (V — 1) dimensional vector whose 
components are riff, withy fixed and i f*j. Also let P(j),j = 1, • ■ •, V, 
be the (V — 1) X (V— 1) matrix obtained by crossing out they'th row 
and they'th column of P. The P(j) still have positive elements, but the 
irreducibility of P implies that not all rows of P(j) can sum to one, 
and so the largest eigenvalue of P(j) is strictly less than one. The 
equations represented by (55) may now be written 

[I-P(j)JN^ = u, y=l, ...,V f (56) 

where u is the (V — 1) dimensional vector having all components 
unity. From (56), 

N<» = [/ _ PU)]-^ = [I+ P(j) + P 2 (j) + . . .]u. (57) 

If Pn(j) denotes the (r, s) element of the Ath power of P(j), then (1) 
and (57) yield 
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V V-l 00 



N= V(V u l I 2 PW), (58) 

V(V — l) y=i r,s-l Af-0 

where by P°(j) we mean the identity matrix for ( V - 1) dimensions. 
Equation (58) succumbs to the application of the following: 
Lemma: Let P be any stochastic V X V matrix and let P{j), j = 
1, • • -, Vbe the (V- 1) X (V- 1) matrix obtained by crossing out the 
jth row andjth column of P. Then 



y y>(/)>( ° for k ~° 

jhrhi '-[V(V-k-l) for 0</z<V-2. 



(59) 



Proof: The positivity of the sum for all k is trivial since P(j) has 
nonnegative elements. If k = 0, P°(j) is the identity for ally, so the 
sum of all its elements if (V - 1), and we obtain V(V — 1) when we 
sum over/. Next consider k = 1 and make use of the stochastic nature 
ofP, that is £. P„ = 1. 

S £ PrsU) = HPrs=lla- Prj) 
j r,s j r+j j ntj 

- V( V - 1) - £ £ P* + l Prr & V( V - 2) . 

/ r r 

Now proceed by induction assuming that the lemma is true for k and 
show it true for (k + 1) 

V V-l 

2 2 p?: 1 

j-\ r,s-\ 



— 2j Z. " n \'*tr "2 "a * " 


* n *+l n *+2 


/ "i»*> 




"*+2'*> 




= 2j 2j Pn 1 n 2 ""2 n a ' ' 


• °"*"* + i 


j "i*/ 




»*+i»*y 




— 2 2 •* n i n 2 * ■ 


' * n k+lJ 


/ «i»*y 




»*+i»V' 




>V(V-*-l)-2 


2 *»l«i 


/ ". 


,,n 2 --- 



Pn u „, (60) 



The inequality follows from the induction assumption (true for £) and 
is further strengthened by extending the range of summation on the 
negative term. In the negative term, the sums over./, »*+i, • • •, zi 2 yield 
unity, and, finally, the sum over n t gives V. The conclusions of the 
lemma follow. 

Now apply the lemma to (59). 
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N * V(V n I ZIPUJ) 

1 v ~ 2 V 

I V(V-k-l)=-, (61) 



-V(V-l) k % 2 

as desired. 

The condition for equality in (61) obviously requires that 

•••P„„, A-l, .... V-l. (62) 

Since all terms on the right member of (62) are nonegative, any term 
present there but not appearing on the left must be zero. In particular, 
we must have for anyy 

Pjj = o if k = 1 (63) 

I Pj^---Pn r j = *-%...,V-l. (64) 

These equations state that it is impossible to return to any initial state 
/ in less than V steps. This, plus irreducibility, implies the unidirec- 
tional movement or a circle. 

The reason why the lemma is exact in this case is that each P(j) is, 
then — except for a reordering of nodes — a canonical Jordan block with 
all zeros in the matrix except for (V — 2) ones on the appropriate off- 
diagonal. 

IV. THE WORST WALK 

We have been unable to describe the worst setup for random routing. 
For V = 3, 4, and 5, numerical work shows that arranging the vertices 
on a straight line gives the worst cases. In fact, for V nodes on a 
straight line it is possible to show that 

V 2 -l 
N = — 3— , (65) 

which is significantly worse than (V — 1), the best attainable with 
random routing. However, for large Vone can do worse than (65). Our 
result for this situation is 

0(y 3 )<AU, re t<0(V 35 ). (66) 

In (66) iVworet denotes the largest value of N obtained from any of the 
connected network graphs on the V vertices. 
For the lower bound, assume V = 6m — 1 and consider the bar- 
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Fig. 1 — An 11 -node barbell. 

bell graphs described by Landau-Odlyzko. 3 This class of graphs has 
(2m — 1) nodes connected in a straight line with complete graphs of 
2m nodes attached to each end of the line by a single edge of the graph. 
If m = 1, this only describes a straight line, but for m = 2 or greater, 
the barbell nature is evident. The case for V = 11 is shown in Fig. 1. 

In Ref. 3 the authors show that for such graphs X 2 , the second largest 
eigenvalue of P = DA, satisfies 



X 2 > 1 - -^ 



(67) 



with Y = 54 + O (i). Then from (26), (67), and (34), 



iV> 



a 2 



> (XFVW 



(68) 



V - 1 1 - X 2 

>0(V 2 )/iv-i, 
jiiv-i being the smallest nonzero eigenvalue of M. Equation (44) shows 

Hv-i>— — ■ (69) 

maxp, 

Since the nodes of the complete graphs in the barbell each have 0( V) 
incident edges and there are 0(V 2 ) edges in the barbell, (12) implies 



(70) 



that maxp, = O ( — I. Equations (68) and (69) then give 
N>0(V 2 )0(V) = 0(V 3 ). 



Although we will not give the details here, it is possible to show that, 
in fact, N = 0(V 3 ) for barbell graphs. 

Our upper bound will be based on (57), but first we need the result 
that the largest eigenvalue of P(j) satisifes 

1 
V 3 



\i(j) = UPU)]^i-0{— 3 



(71) 



Just as the matrix P(j) was defined in Section III, now introduce 
matrices D(j) and A(j) by eliminating they'th row and column of D 
and A, respectively. Then, 

P(j)=D{j)A(j). (72) 
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Also set 

Q(j) = D l/2 (j)Mj)D 1/2 U) (73) 

so that Q(j) is symmetric and has the same eigenvalues as P(j). In 
fact, we have 

Xi[PO)] = UQU)] = mp y + Q(y)y 

= max z t A(y')z = max xUx. (74) 

Note that in (74) we have returned to the V X V adjacency matrix A 
by introducing the constraint Xj = 0. If S is the set of those (ordered) 
pairs of indices which corresponds to vertices connected by an edge of 
the graph, we have + 

X AX. = 2j X m X n = — 2j LX"> "^ %n — \X m — X n ) J 
(m,n)(ES 6 S 

= £ ImXlr ~ - J (X m ~ Xnf. (75) 

m * S 

Using (75) in (74) we easily obtain 

My) - 1 - = min £ (* m - x n ) 2 . (76) 

2 xj-o s 

Let x* be the component of x having the largest square. Then 

1-EWS(V-1)UL (77) 

or 

* |a -U^!j' <78 > 

where / max (or /min) is the maximum (or minimum) of the k, i = 

u—.v. 

Now, from the connectivity of the graph, vertex j is attached to 
another vertex t. So, using (76), 

Mj) < 1 - x 2 ti (79) 

since xj = 0.Ift = k, (78) and (79) yield 

My) ^ i - , T/ _ 1 1W (t = k). (80) 

If t ¥> k, there exists a chain of r distinct edges joing vertices t and k. 



f The remainder of this demonstration is entirely inspired by Ref. 3. 
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Clearly, r < d, where d is the diameter of the graph. Then, using the 
basic trick of Ref. 3, 

x k - x,= (x k - x k ) + (x kl - x*2> + • • • + (*Vi - *<) ( 81 ) 

and so, by Cauchy's inequality, 

(Xk ~ X t ? < £ £ {X m - X n ) 2 < ^ J <*„ - *„) 2 . (82) 

Equations (76) and (82) thus yield 

M7):£l- ( **~*' )2 . (83) 

a 

Combine (79) and (83) by averaging and then minimize over the 
numerical value of x t to obtain, with (78), for t ¥> k, 



x,0)<i-i 



2 (x k - x t y 

Xt + ; 



<1 - . (84) 

2(l + d)(V-Mn« 



Both cases (80) and (84) are included in 

M J )sl - 2a + d)tv-i ) i^ 1 -W" (85) 

since d<(V- 1) and Z max < (V- 1). 

To complete the upper bound on N, start with (58) and write 
(suppressing the y'-dependence of the matrices in the notation) 



ViV-l)'] 



= T r l\\D l, V-QVDV 2 \\ 
" j 

^^Eii5 i/2 iiii^- i/2 iiii[/-Qrii 

(lT i v i 

/ 'max - 1 ^ * 



(86) 



From Zmin > 1, J ma x < ( V - 1), (85) and (86) we thus have 

N < 2VVV-1, (87) 

as desired. 

Is it possible that the complete graph is optimal for random routing 
because adding any edge to a graph decreases AT? No. Many edges 
must be added to the straight-line graph to form the barbell, but the 
former has N = 0(V 2 ) while for the latter, N = 0(V 3 ). 

MARKOV CHAINS 2079 



REFERENCES 

1. L. Kleinrock, Communication Nets, New York: McGraw-Hill, 1964, Chap. 6. 

2. D. Mitra and A. Weiss, "Analysis of Delay-Differential Equations Arising in Com- 

munication Network Synchronization," in Proc. IEEE Int. Symp. Circuits and 
Systems, Houston, Texas, 1980, pp. 839-43. 

3. H. J. Landau and A. M. Odlyzko, "Bounds for Eigenvalues of Certain Stochastic 

Matrices," Linear Algebra and Applications, 38 (June 1981), pp. 5-15. 

4. W. Feller, An Introduction to Probability Theory and Its Applications, New York: 

John Wiley, 1957, 2nd ed. 

5. E. F. Beckenbach and R. Bellman, Inequalities, New York: Springer- Verlag, 1965, 

p. 76, Theorem 24. 

6. Peter Lancaster, Theory of Matrices, New York: Academic Press, 1969. 



2080 THE BELL SYSTEM TECHNICAL JOURNAL, OCTOBER 1 982 



